Skip to content

HYPERFLEET-1132 - feat: enable parallel E2E execution via Ginkgo#150

Open
kuudori wants to merge 1 commit into
openshift-hyperfleet:mainfrom
kuudori:parallel-e2e-phase0-view
Open

HYPERFLEET-1132 - feat: enable parallel E2E execution via Ginkgo#150
kuudori wants to merge 1 commit into
openshift-hyperfleet:mainfrom
kuudori:parallel-e2e-phase0-view

Conversation

@kuudori

@kuudori kuudori commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Phase 0 preparation for Ginkgo parallel E2E execution (--procs=N). All changes are inert at 1 process - no behavioral change.

Suite hooks:

  • Convert AfterSuite to SynchronizedAfterSuite so that K8s resource sweep runs exactly once (after all processes finish) while Pub/Sub cleanup runs per-process (each process tracks its own deployments).
  • Split CleanupResources into CleanupPubSubResources (per-process) and CleanupKubeResources (final, once). Original CleanupResources kept as convenience wrapper for single-process mode.

Serial labels added to specs that are unsafe under concurrency:

  • 9 perf specs: latency thresholds calibrated on a quiet system
  • Re-DELETE idempotency: scales sentinel to 0 replicas
  • 3 temp-adapter specs: deploy adapters subscribing to all events
  • Comments added to all Serial decorators (including pre-existing ones)

Documentation:

  • Add Parallel Safety section to docs/development.md with rules for writing parallel-safe tests and valid reasons for Serial.

Summary

  • HYPERFLEET-XXX

Test Plan

  • Unit tests added/updated
  • make test-all passes
  • make lint passes
  • Helm chart changes validated with make test-helm (if applicable)
  • Deployed to a development cluster and verified (if Helm/config changes)
  • E2E tests passed (if cross-component or major changes)

@openshift-ci
openshift-ci Bot requested review from aredenba-rh and rh-amarin July 17, 2026 19:39
@openshift-ci

openshift-ci Bot commented Jul 17, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign kuudori for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@coderabbitai

coderabbitai Bot commented Jul 17, 2026

Copy link
Copy Markdown

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Central YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: ddfc4d36-387d-4f0c-9d2c-fddcdc985f07

📥 Commits

Reviewing files that changed from the base of the PR and between d3561e8 and 8756517.

⛔ Files ignored due to path filters (1)
  • .bingo/ginkgo.sum is excluded by !**/*.sum
📒 Files selected for processing (16)
  • .bingo/Variables.mk
  • .bingo/ginkgo.mod
  • .bingo/variables.env
  • AGENTS.md
  • Dockerfile
  • Makefile
  • docs/development.md
  • e2e/adapter/maestro_unavailability.go
  • e2e/cluster/adapter_failure.go
  • e2e/cluster/crash_recovery.go
  • e2e/cluster/delete_edge_cases.go
  • e2e/cluster/stuck_deletion.go
  • e2e/e2e_suite_test.go
  • e2e/nodepool/creation.go
  • pkg/e2e/suite.go
  • pkg/helper/cleanup.go
🔗 Linked repositories identified

CodeRabbit considers these linked repositories for cross-repo context during reviews:

  • openshift-hyperfleet/architecture (manual)
  • openshift-hyperfleet/hyperfleet-api (manual)
  • openshift-hyperfleet/hyperfleet-sentinel (manual)
  • openshift-hyperfleet/hyperfleet-adapter (manual)
  • openshift-hyperfleet/hyperfleet-broker (manual)
🚧 Files skipped from review as they are similar to previous changes (6)
  • e2e/cluster/crash_recovery.go
  • e2e/cluster/delete_edge_cases.go
  • pkg/e2e/suite.go
  • e2e/cluster/adapter_failure.go
  • docs/development.md
  • pkg/helper/cleanup.go

📝 Walkthrough

Summary by CodeRabbit

  • New Features

    • E2E tests can now run in parallel, with configurable worker counts and improved test reporting.
    • Added a dedicated E2E test entry point and packaged test tools for streamlined execution.
  • Bug Fixes

    • Improved cleanup reliability during parallel test runs by coordinating shared resource removal.
  • Documentation

    • Added guidance for parallel-safe tests, serial execution, performance tests, and E2E file naming.
  • Refactor

    • Updated selected tests to run in a controlled order when they require shared infrastructure.

Walkthrough

The E2E workflow now uses a pinned Ginkgo CLI and compiled test binary, with configurable multi-process execution through --procs. A new test entrypoint loads suite configuration and runs Ginkgo. Documentation defines parallel-safety rules and serial-suite criteria. Several suites now use Serial or Ordered lifecycle controls. Teardown uses synchronized cleanup: each process handles Pub/Sub resources, while process 1 cleans Kubernetes and Helm resources and clears suite state.

Sequence Diagram(s)

sequenceDiagram
  participant CI
  participant Ginkgo
  participant E2EProcesses
  participant Cleanup
  CI->>Ginkgo: invoke E2E suite with --procs
  Ginkgo->>E2EProcesses: distribute specs across processes
  E2EProcesses->>Cleanup: perform per-process Pub/Sub cleanup
  E2EProcesses->>Cleanup: process 1 performs Kubernetes and Helm cleanup
Loading

Estimated code review effort: 3 (Moderate) | ~25 minutes

Suggested reviewers: aredenba-rh, rh-amarin

🚥 Pre-merge checks | ✅ 4 | ❌ 7

❌ Failed checks (1 warning, 6 inconclusive)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 60.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
Sec-02: Secrets In Log Output ❓ Inconclusive Repository clone failed, so this custom check could not run with code access. Retry the review run. If this persists, inspect pre-merge custom-check logs for infrastructure or agent runtime failures.
No Hardcoded Secrets ❓ Inconclusive Repository clone failed, so this custom check could not run with code access. Retry the review run. If this persists, inspect pre-merge custom-check logs for infrastructure or agent runtime failures.
No Weak Cryptography ❓ Inconclusive Repository clone failed, so this custom check could not run with code access. Retry the review run. If this persists, inspect pre-merge custom-check logs for infrastructure or agent runtime failures.
No Injection Vectors ❓ Inconclusive Repository clone failed, so this custom check could not run with code access. Retry the review run. If this persists, inspect pre-merge custom-check logs for infrastructure or agent runtime failures.
No Privileged Containers ❓ Inconclusive Repository clone failed, so this custom check could not run with code access. Retry the review run. If this persists, inspect pre-merge custom-check logs for infrastructure or agent runtime failures.
No Pii Or Sensitive Data In Logs ❓ Inconclusive Repository clone failed, so this custom check could not run with code access. Retry the review run. If this persists, inspect pre-merge custom-check logs for infrastructure or agent runtime failures.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title matches the PR’s main change: enabling parallel E2E execution via Ginkgo.
Description check ✅ Passed The description is directly related to the changeset and accurately covers the cleanup, Serial, and docs updates.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
✨ Simplify code
  • Create PR with simplified code

Warning

Review ran into problems

🔥 Problems

Repository analysis: Couldn't refresh openshift-hyperfleet/hyperfleet-e2e clone - clone failed: Clone operation failed: Stream initialization permanently failed: 14 UNAVAILABLE: read ECONNRESET


Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pkg/helper/cleanup.go`:
- Around line 78-95: Update CleanupPubSubResources to construct its helper
through a Pub/Sub-specific constructor that does not create Kubernetes or
dynamic clients. Add NewPubSubCleanupHelper alongside NewCleanupHelper,
initializing only cfg and adapterDeploymentList with the same validation, then
use it for SweepPubsubTestAdapterResources so Kubernetes client failures cannot
block Pub/Sub cleanup.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Central YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: f49c0414-fb75-48f5-9888-4d902a547296

📥 Commits

Reviewing files that changed from the base of the PR and between 4f0fac7 and d3561e8.

📒 Files selected for processing (20)
  • docs/development.md
  • e2e/adapter/adapter_failover.go
  • e2e/adapter/adapter_with_maestro.go
  • e2e/adapter/maestro_unavailability.go
  • e2e/cluster/adapter_failure.go
  • e2e/cluster/crash_recovery.go
  • e2e/cluster/delete_edge_cases.go
  • e2e/cluster/force_delete.go
  • e2e/cluster/perf_cascade_delete_latency.go
  • e2e/cluster/perf_create_latency.go
  • e2e/cluster/perf_delete_latency.go
  • e2e/cluster/perf_list_filtered_latency.go
  • e2e/cluster/perf_list_latency.go
  • e2e/cluster/perf_read_entity_size_latency.go
  • e2e/cluster/perf_update_latency.go
  • e2e/cluster/stuck_deletion.go
  • e2e/nodepool/perf_create_latency.go
  • e2e/nodepool/perf_delete_latency.go
  • pkg/e2e/suite.go
  • pkg/helper/cleanup.go
🔗 Linked repositories identified

CodeRabbit considers these linked repositories for cross-repo context during reviews:

  • openshift-hyperfleet/architecture (manual)
  • openshift-hyperfleet/hyperfleet-api (manual)
  • openshift-hyperfleet/hyperfleet-sentinel (manual)
  • openshift-hyperfleet/hyperfleet-adapter (manual)
  • openshift-hyperfleet/hyperfleet-broker (manual)

Comment thread pkg/helper/cleanup.go
@kuudori
kuudori marked this pull request as draft July 18, 2026 00:18
@kuudori
kuudori force-pushed the parallel-e2e-phase0-view branch 3 times, most recently from 008183f to 49a425e Compare July 21, 2026 14:04
@kuudori kuudori changed the title test: parallel-safety hygiene for future multi-process execution HYPERFLEET-1132 - feat: enable parallel E2E execution via Ginkgo Jul 21, 2026
- Add ginkgo CLI binary (v2.27.2) via .bingo tooling
- Add e2e_suite_test.go entry point for ginkgo --procs=N
- Add e2e-ginkgo Makefile target with PROCS= support
- Build ginkgo CLI and e2e.test binary in Dockerfile
- Add ginkgo.Serial to 5 infrastructure-modifying tests
- Convert nodepool creation to Ordered+BeforeAll for isolation
- Split AfterSuite into SynchronizedAfterSuite for multi-process safety
- Split CleanupResources into per-process and process-1-only phases
- Add parallel safety documentation
@kuudori
kuudori force-pushed the parallel-e2e-phase0-view branch from 49a425e to 8756517 Compare July 21, 2026 15:37
@kuudori
kuudori marked this pull request as ready for review July 21, 2026 15:38
@openshift-ci
openshift-ci Bot requested a review from rafabene July 21, 2026 15:39
@kuudori

kuudori commented Jul 21, 2026

Copy link
Copy Markdown
Contributor Author

@coderabbitai review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant